1. Field of the Invention
The present invention generally relates to computer systems, and more particularly to a memory hierarchy for a computer system that includes large cache structures having different latencies across the cache arrays.
2. Description of the Related Art
The basic structure of a conventional computer system includes one or more processing units which are connected to various peripheral devices (including input/output devices such as a display monitor, keyboard, and permanent storage device), a memory device such as random access memory (RAM) that is used by the processing units to carry out program instructions and store operand data, and firmware which seeks out and loads an operating system from one of the peripherals (usually the permanent memory device) whenever the computer is first turned on. The processing units typically communicate with the peripheral devices by means of a generalized interconnect or bus. A computer system may have many additional components such as various adapters or controllers, and serial, parallel and universal bus ports for connection to, e.g., modems, printers or network interfaces.
In a symmetric multi-processor (SMP) computer, all of the processing units are generally identical, that is, they all use a common set or subset of instructions and protocols to operate, and generally have the same architecture. A typical architecture includes a processor core having a plurality of registers and execution units, which carry out program instructions in order to operate the computer. The processing unit can also have one or more caches, such as an instruction cache and a data cache, which are implemented using high speed memory devices. Caches are commonly used to temporarily store values that might be repeatedly accessed by a processor, in order to speed up performance by avoiding the longer step of loading the values from a main memory device. These caches are referred to as “on-board” when they are integrally packaged with the processor core on a single integrated chip.
A processing unit can include additional caches, such as a level 2 (L2) cache which may support on-board (level 1) instruction and data caches. An L2 cache acts as an intermediary between the main (system) memory and the on-board caches, and can store a much larger amount of information than the on-board caches, but at a longer access penalty.
A cache has many blocks which individually store the various instruction and data values. The blocks in any cache are divided into groups of blocks called sets or congruence classes. A set is the collection of cache blocks that a given memory block can reside in. For any given memory block, there is a unique set in the cache that the block can be mapped into, according to preset mapping functions. The number of blocks in a set is referred to as the associativity of the cache, e.g. 2-way set associative means that for any given memory block there are two blocks in the cache that the memory block can be mapped into; however, several different blocks in main memory can be mapped to any given set. A 1-way set associative cache is direct mapped, that is, there is only one cache block that can contain a particular memory block. A cache is said to be fully associative if a memory block can occupy any cache block, i.e., there is one congruence class, and the address tag is the full address of the memory block.
An exemplary cache line (block) includes an address tag field, a state bit field, an inclusivity bit field, and a value field for storing the actual instruction or data. The state bit field and inclusivity bit fields are used to maintain cache coherency in a multiprocessor computer system (to indicate the validity of the value stored in the cache). The address tag is usually a subset of the full address of the corresponding memory block. A compare match of an incoming address with one of the tags within the address tag field indicates a cache “hit.” The collection of all of the address tags in a cache (and sometimes the state bit and inclusivity bit fields) is referred to as a directory, and the collection of all of the value fields is the cache entry array.
When all of the blocks in a congruence class for a given cache are full and that cache receives a request, whether a “read” or “write,” to a memory location that maps into the full congruence class, the cache must make one of the blocks in that class available for the new operation. The cache chooses a block by one of a number of means known to those skilled in the art (least recently used (LRU), random, pseudo-LRU, etc.). If the data in the chosen block has been modified, that data is written to the next lowest level in the memory hierarchy which may be another cache (in the case of the L1 or on-board cache) or main memory (in the case of an L2 cache). By the principle of inclusion, the lower level of the hierarchy will already have a block available to hold the written modified data. If the data in the chosen block has not been modified, the value in that block is simply abandoned and not written to the next lowest level in the hierarchy. This process of freeing up a block from one level of the cache hierarchy is known as an eviction. At the end of this process, the cache no longer holds a copy of the evicted block. When a device such as the CPU or system bus needs to know if a particular cache line is located in a given cache, it can perform a “snoop” request to see if the address is in the directory for that cache.
As microprocessor computing power grows, it becomes more critical for caches to correspondingly grow in size in order to avoid processing bottlenecks that arise from memory latencies. However, large cache structures can introduce or exacerbate other problems, such as bandwidth and connectivity. Some high-performance computer systems address these issues by dividing the cache array and directory into two or more slices, and allowing multiple access/command ports. One example of such a sliced cache structure is shown in FIG. 1, which depicts a processing unit 10 having a processor core 12 with on-board instruction and data caches, and an L2 cache entry array which is divided into two slices 14a and 14b (slice A and slice B). The L2 cache controller is divided into two corresponding slices 16a, 16b each having its own directory 18a, 18b. When processor core 12 issues a load request, the address tag for the request is sent to one of the directory slices 18a, 18b, based on a hash scheme that uses an address bit to direct the request to a given slice (e.g., addr(56)=0 means slice A). The L2 directory slice performs the address comparisons and upon detecting a load hit activates a select signal that controls the output of cache array slices 14a and 14b. The “addr.rw.ws” signal includes information regarding the congruence class for the requested memory block, whether the operation is a read or write, and the write set.
Each cache array slice 14a, 14b is further divided into four sectors, that is, a given cache line is distributed across all four sectors of a slice. In this example, each cache line is 128 bytes longs, and the digit pairs in each sector represent the beginning byte number (in hexadecimal) for an 8-byte word of the line, e.g., “00” refers to the first 8-byte word in the cache line (bytes 00, 01, 02, 03, 04, 05, 06 and 07), and “08” refers to the second 8-byte word in the cache line (bytes 08, 09, 0A, 0B, 0C, 0D, 0E and 0F). Thus, each sector contains 32 noncontiguous bytes of a given cache line. All of the sectors are in a single latency domain but only 32 bytes are output in a given cycle, so it takes four cycles to output a complete 128-byte cache line, with the entire cache array (all sectors) powered up during each of the four cycles.
Each L2 controller slice 16a, 16b has its own read claim (RC), cast out (CO) and snoop (SN) machines. Each controller slice further has its own directory arbiter 20a, 20b which handles conflicts between these machines and load requests from the processor core. The directory arbiters are connected respectively to cache arbiters 22a, 22b which control the flow of merge data coming from elsewhere in the memory hierarchy (e.g., system memory) using separate command ports. Merge flow logic in each cache slice receives 32 bytes in a given cycle from four 8-byte fabric busses that are connected to system memory and various peripheral devices.
While the use of sliced cache arrays can improve cache bandwidth, there are still serious problems with power consumption, wiring topology, differential latencies, and recoverability, especially when the design scales to larger cache sizes. As designs grow the cache size by placing larger numbers of cache array macros, the latency to the farthest array becomes multiple clock cycles away from the core compared to the closest cache array. Thus, the prior art mechanism wherein all arrays' access times are in the same clock cycle becomes temporally wasteful, because the close arrays must be slowed to match the farthest arrays' access time. Although transmission speed can be increased by providing special wiring (wider/faster), such wiring increases the expense of the design and uses valuable wiring resources, and these problems are compounded in designs requiring large busses for two cache slices. Even in the case of a load hit, there can still be a significant delay in accessing and transmitting the requested cache line, due to the physical layout of the cache and processor core. It would, therefore, be desirable to devise an improved cache structure which could reduce latencies associated with a sizeable growth of the cache, particularly latencies arising from load hits. It would be further advantageous if the cache structure could maintain superior directory bandwidth, and still afford a high degree of recoverability in the case of a defect in the array.